Virtually all published
database benchmark figures are now either TPC-C or TPC-D test results. Database
vendors know the TPC tests well and participate in TPC testing on an ongoing
basis. So why another benchmark?
Because TPC tests measure the
combined performance of a database, operating system, transaction monitor and
hardware platform, it’s very hard to make accurate database-to-database
comparisons from TPC numbers. By tightly controlling our test system, we will
gain a far clearer picture of how the databases themselves perform, independent
of operating system, TP monitor and hardware.
This test is designed to be
between TPC-C and TPC-D in its mix of OLTP and DSS queries—we have broader DSS
coverage than TPC-C while still keeping a predominantly OLTP flavor.
As with TPC-C, the core of
our test is a read/write OLTP mix. However, we also have a large ad hoc DSS
test mix and load, index and export tests (just as in TPC-D) because these
operations are so important for warehousing.
We feel if medium-size
organizations only have one database server, our mix is a reasonable
approximation of their workload. Neither TPC-C nor TPC-D has this type of mixed
workload testing.
TPC testing is done over long
periods of time with all the in-house experts a database vendor has available.
In addition, sizing requirements require the use of hardware configurations
that most corporations would never buy (tens of thousands of users, more than
500GB of disk space, and so on). As such, TPC metrics are a good measure of
absolute performance but a poor measure of typical performance.
In contrast, we are only
interested in relative results, and will firmly resist what we feel are
extraordinary tuning efforts (as outlined in Section 11). Our goal is to model
how the tested databases perform on mainstream (though top-of-the-line)
hardware, in a production, mixed workload environment managed by a busy DBA.
Our test environment will be
much more tightly controlled than in TPC tests in order to maximize
comparability, reproducibility and speed at which we can test new products. All
vendors will be tested using the same hardware and, as much as possible, using
the same operating system configuration.
We have outstanding, fully
automated, test control logic designed specifically for this kind of database
testing and have designed our test suite to be relatively simple for a vendor
to implement. With proper preparation, we feel a full test can be accomplished
in one week (though we will be allowing two weeks for our first run-through to
make doubly sure the process is smooth).
Like most high-end database
benchmarks, our data generator is specifically designed to maintain constant
data set properties (cardinality and distribution) at a number of different
data set sizes. In addition, we have designed the benchmark operations to be
easily ported to different database platforms. These characteristics will allow
us to design a modified version of this test for desktop databases, and it our
plan to do just that.
As described above, this
benchmark models a mixed workload OLTP database environment. Though we have
included a DSS component to this test, this is not an OLAP benchmark. We’re
quite interested in performance testing dedicated warehousing products like Red
Brick Warehouse or Sybase IQ and are investigating the various competing
proposals in this space from the OLAP Council, Red Brick, MicroStrategy and the
TPC.
This is not a SQL compliance
test and we are not verifying SQL-92 or SQL3 compatibility. (We will, however,
be running data consistency checks during our testing to ensure the database
under test implements ACID transactions.)
This is not a clustered or
distributed database benchmark. There is no requirement for two phase commit
capabilities and all transactions and data are located on a single server.
This is not a platform for
demonstrating the advanced features of your database. While we definitely want
you to optimize your installation for optimal performance, our test uses only
mainstream SQL (along with commonly supported features like stored procedures)
and can be implemented by all the major relational database vendors on the
market.
We have future plans for
special benchmarks specifically designed to test advanced features found in
SQL3 (for example, comparing the performance of object pointers to joins),
replication speed and the performance impact of the various extensibility
models on the market.